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SUMMARY. 

Quantitative  procedures  were  developed  for  testing  block-structured  models  for  multi¬ 
input  nonlinear  visual  circuits  studied  with  spatiotemporal  white  noise.  A  linear-nonlinear  (LN) 
model  test  index  was  found  to  be  suitable  for  classifying  cells  as  simple  versus  complex.  Although 
simple  cells  were  better  modeled  as  LN  systems  than  complex  cells,  most  simple  cells  deviated 
considerably  from  LN  behavior.  A  nonlinearity  of  cortical  origin  would  appear  to  be  responsible, 
possibly  activated  more  strongly  by  broadband  noise  than  by  sinewave  grating  stimuli.  Also, 
two  classes  of  binocular  complex  cells  were  identified.  Whereas  all  binocular  complex  cells 
necessarily  have  a  non-zero  second-order  “same-eye’  interaction  kernel,  their  second-order  “cross- 
eye”  interaction  kernel  could,  it  was  found,  be  either  non-zero  or  identically  zero.  The  two 
binocular  complex  c  classes  are  thought  to  be  the  monkey  equivalents  of  the  “phase  specific” 
and  “phase  non-specific”  cells  previously  discovered  by  Ohzawa  and  FYeeman  (1986b)  in  the 
cat.  A  random-jitter  disparity  tuning  model  suggested  by  Ohzawa  and  Freeman  was  found 
to  be  inconsistent  with  “phase  non-specific”  cells  in  the  monkey,  rather  they  must  linearly 
combine  nonlinear  monocular  inputs — as  alternatively  suggested  by  Ohzawa  and  FVeeman.  The 
left  and  right  eye  filters  of  simple  cells  in  the  monkey  were  found  to  have  broadly  varying  phase 
relationships.  Finally,  a  quadrature  correlation  model  for  stereo  depth  estimation  was  developed 
to  account  for  recent  new  findings  by  ourselves,  and  others. 

RESEARCH  OBJECTIVES. 

The  major  objectives  of  the  project  were  as  follows: 

I.  Develop  quantitative  procedures  for  testing  block-structured  models  for  multi¬ 
input  nonlinear  visual  circuits.  These  methods  would  be  used  in  the  fiilliillment  of  further 
objectives  described  below. 

II.  Determine  if  binocular  simple  cells  in  the  monkey  are  well  modelled  by  a  spa¬ 
tiotemporal  linear- nonlinear  (LN)  csiscade  model.  Previous  work,  reviewed  in  Pollen, 
Jacobson  and  Gaska  (1989),  had  long  established  that  monocular  simple  cells  in  the  cat  and 
monkey  could  be  well  modelled  by  a  linear  filter  followed  by  a  threshold  nonlinearity.  Ohzawa 
and  FYeeman  (1986a)  had  recently  extended  these  results  in  the  cat  by  observing  that  binocular 
simple  cells  in  the  cat  combine  left-  and  right-eye  inputs  linearly  prior  to  response  truncation. 
The  techniques  developed  in  objective  1  were  to  be  used  to  similarly  evaluate  the  LN  model  for 
binocular  simple  cells  in  the  monkey. 

III.  Determine  if  the  two  classes  of  binocular  complex  cells,  ‘^phase-specific**  and 
“phase  non-specific,**  previously  identified  by  Ohzawa  and  FVeeman  (1986b)  in  the 
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cat  cortex,  are  also  present  in  the  monkey  cortex.  These  two  complex  cell  classes  were 
distinguished  by  their  sensitivity  (or  lack  thereof)  to  the  relative  phase  of  a  pair  of  dichoptically 
presented  spatial  sinewave  gratings.  If  these  two  binocular  cell  classes  were  also  found  to  exist 
in  the  monkey,  we  wished  to  obtain  improved  models  of  these  cells  as  discussed  below. 

rV.  Determine  if  **phase-8peciflc**  complex  cells  in  the  monkey  can  be  well  modelled 
by  a  straightforward  binocular  generalization  of  the  Pollen  and  Ronner  (1981)  4-cell 
quadrature  model,  where  the  phase-specific  complex  cell  would  receive  inputs  from  four  binoc¬ 
ular  simple  cells  whose  phase  relation  is  quadrature  with  respect  to  each  eye  individually.  Such 
a  phase-specific  complex-cell  model  had  been  previously  proposed  by  the  investigators  (Gaska 
et  ai,  1988),  (and  implemented  by  them  in  a  system  for  estimating  disparity),  and  essentially 
the  saune  4-cell  binocular  model  was  later  independently  proposed  by  Ohzawa,  DeAngelis  and 
Freeman,  (1990). 

V.  Determine  if  **phase  non-specific**  cells  in  the  monkey  are  better  modelled  (a) 
by  assuming  that  they  sum  two  or  more  monocular  complex  cells — each  sensitive 
to  inputs  from  a  different  eye,  or  rather,  (b)  by  assuming  that  their  insensitivity 
to  the  relative  phaise  of  extended  dichoptic  gratings  results  from  a  random  jitter  of 
binocular  disparity  tuning  throughout  the  receptive  field.  Both  types  of  models  had 
been  proposed  by  Ohzawa  and  Freeman  (1986b)  to  be  consistent  with  their  data  from  the  cat. 
Experiments  to  differentiate  between  these  models  had  not  yet  been  performed  in  either  the  cat 
or  monkey  species. 

VI.  Determine  how  the  phase  differences  of  the  left-  and  right-eye  spatial  filters 
of  simple  cells  in  the  monkey  are  distributed.  In  particular,  do  the  phases  of  the  left- 
and  right-eye  spatial  filters  of  simple  cells  differ  by  0  and  180  degrees  exclusively,  as  in  the 
model  previously  proposed  by  the  investigators  (Gaska  et  ai,  1988),  or  do  the  phase  differences 
distribute  more  broadly. 

VII.  Incorporate  major  new  physiological  findings  into  a  model,  previously  proposed 
by  the  investigators,  for  binocular  simple  and  complex  cells  and  their  respective 
roles  in  disparity  estimation. 
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RESEARCH  STATUS 


New  systems  analysis  methods 

In  this  section  we  describe  results  we  have  developed,  under  this  contract,  for  testing 
block-structured  models  for  multi-input  nonlinear  visual  circuits.  These  new  methods  were 
developed  subject  to  the  following  important  constraints  and  objectives: 

1)  The  specific  block>structured  models  of  interest  could  be  described  by 
an  interconnection  of  (a)  linear  time-invariant  filters,  and  (b)  memoryless  (static) 
nonlinearities.  Most  monocular  and  binocular  models  that  have  been  proposed  for  simple  and 
complex  cells  fall  into  the  above  class  of  block-structured  models.  For  example,  consider  a  linear- 
nonlinear  (LN)  cascade  system  that  operates  on  spatiotemporal  inputs  to  produce  a  temporal 
output.  Such  a  model  is  commonly  attributed  to  simple  cells  in  the  visual  cortex,  where  L  is  the 
spatiotemporal  receptive  field  profile  of  the  cell,  and  N  is  the  threshold  nonlinearity  that  relates 
the  cell’s  output  spike  rate  to  its  somal  potential. 

2)  The  structural  model  testing  procedures  were  to  be  based  solely  on  ex¬ 
perimental  stimulus-response  data.  Such  data  would  generally  be  obtained  during  electro 
physiolojpcal  recording  experiments  which  seek  to  characterize  the  input-output  relationship  of 
a  visual  neuron. 

3)  It  was  assumed  that  the  multi-input  stimuli  used  to  evoke  photic  responses 
from  visual  neurons  would  be  computer-generated  approximations  to  spatiotempo¬ 
ral  white  noise.  Such  a  stimulus  generally  consists  of  multiple  spatial  regions  (bars,  squares, 
etc.)  whose  luminances  are  varied  randomly  and  independently  as  a  function  of  time.  Spa¬ 
tiotemporal  white  noise  stimuli  are  relatively  easy  to  generate,  and  there  are  well  developed 
analytic  methods  for  summarizing  the  measured  white  noise  stimulus-response  relation  of  a 
time-invariant  nonlinear  systems  as  a  set  of  functional  expansion  kernels.  For  such  reasons, 
many  of  the  major  laboratories  studying  cortical  vision  (Emerson  at  U.  Rochester,  Freeman  at 
Berkeley,  Palmer  at  U.  Penn,  Pollen  at  U.  Mass.  Med.  (our  group),  Shapley  at  NYU)  have 
utilized  white  noise  stimuli  in  recent  experiments. 

4)  To  simplify  their  application,  the  procedures  for  testing  block-structured 
models  would  be  designed  so  as  not  to  depend  on  the  parameterization  of  the 
multi-input  nonlinear  visual  circuit  under  study.  This  would  therefore  enable  one  to 
test,  with  a  single  application  of  the  procedure,  a  large  class  of  circuits  whose  members  have  a 
common  structural  design,  but  an  infinity  of  parameterizations.  Such  a  class  of  systems  with 
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a  common  structure  is  illustrated  by  the  set  of  all  linear-nonlinear  (LN)  cascade  systems  with 
spatiotemporal  inputs.  As  mentioned  above,  such  a  model  is  commonly  attributed  to  simple 
cells.  Using  the  structural  testing  procedures  to  be  developed  in  this  first  objective,  we  hoped, 
for  example,  to  be  able  to  test  simple  cells  for  consistency  with  a  spatiotemporal  LN  model 
without  regard  for  the  shape  of  the  cell’s  linear  spatiotemporal  filter  or  the  form  of  its  output 
nonlinearity.  Such  a  parameter  independent  LN  test  would  be  analogous  to  using  the  principal 
of  superposition  to  test  a  system  for  linearity.  Namely,  the  superposition  test  for  consistency 
with  the  ‘L’  structural  class  does  not  presuppose  any  specific  parametric  form  for  the  linear  filter 
of  the  system  under  test. 

Overview  of  results.  During  the  present  three-year  contract,  we  have  developed  nonlinear 
structure  classification  and  parameter  identification  methods  that  are  applicable  to  a  broad  class 
of  multi-input  /  single-output  (MISO)  neural  networks  of  relevance  to  cortical  visual  processing 
(Chen,  Jacobson,  and  Gaska,  1990;  Chen,  Jacobson,  Gaska  and  Pollen,  1989).  These  methods 
are  applicable  to  data  from  stimulus-response  studies  that  use  white  noise  stimuli  consisting 
of  multiple  independent  spatial  inputs  including  random  bar  and  random  checkerboard  stimuli. 
Given  the  growing  use  of  such  multi-input  stimuli,  the  methods  discussed  in  the  above  papers 
are  likely  to  find  wide  application  during  coming  years.  The  new  analysis  methods  are  fairly 
easy  to  apply  to  multi-input  neural  systems  as  illustrated  by  our  recent  paper  submitted  to 
Vision  Research  (Jacobson,  Gaska,  Chen  and  Pollen,  1992). 

The  new  methods  employ  the  low-order  kernels  of  a  multi-input  system  which  can  be 
estimated  using  standard  approaches  (Marmarelis  and  Marmarelis,  1978).  Properties  of  the 
individual  kernels  and  their  mathematical  relationship  are  exploited  in  a  fashion  that  largely 
avoids  dependence  on  the  parameterization  of  the  nonlinear  system  under  study.  All  of  our 
objectives  on  this  task  were  met  or  exceeded. 

Prior  work.  The  approach  we  have  developed  for  analyzing  data  from  multi-input  studies 
builds  upon  previous  work  that  was  concerned  with  testing  and  identifying  single-input  systems 
(Korenberg,  1973a, b;  Billings  and  Fakhouri,  1978;  Marmarelis  and  Marmarelis,  1978;  Sh^ley 
and  Victor,  1980;  Victor  and  Shapley,  1980).  These  earlier  methods  made  use  of  estimated 
Wiener  (or  Wiener-like)  self-kernels  to  test  and  parameterize  LN  and  LNL  models  for  systems 
investigated  with  single-input  stimuli  (e.g.,  luminance  modulation  of  a  uniform  spatial  field  or 
a  single  spot  of  light,  or  contrast  modulation  of  a  single  spatial  sinewave  grating).  Korenberg 
(1985)  had  also  developed  an  important  result  for  multi-input  LN  and  LNL  systems;  namely,  he 
established  that  existing  single-input  methods  could  be  applied  separately  to  each  spatial  input 
in  a  multi-input  stimulation  study. 
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New  Methods.  The  new  methods  we  have  developed  for  testing  multi-input  nonlinear  systems 
are  described  in  a  paper  that  appeared  in  Biolo^cal  Cybernetics  (Chen,  Jacobson  and  Gaska, 
1990),  and  further  results  are  described  in  sm  IEEE  conference  paper  (Chen,  Jacobson,  Gaska 
and  Pollen,  1989).  For  details  on  the  new  methods  refer  to  these  two  papers. 

The  prior  work  described  above  was  formally  extended  by  us  in  two  principal  ways  as 
discussed  below: 

(1)  Systems  with  parallel  nonlinear  channels:  First,  we  enlarged  the  class  of  identifiable  system 
structures  to  include  multi-input  models  with  parallel  LNL  paths.  These  include,  for  example, 
the  classic  feedforward  complex  cell  models  in  which  a  complex  cell  summates  the  outputs 
from  multiple  simple-like  mechanisms  that  are  themselves  modeled  as  multi-input  LN  systems. 
The  extensions  also  encompass  the  half-phase  and  full-phase  Reichardt  motion  detection  models 
(Reichardt,  1961).  We  have  also  developed  multi-input  extensions,  yet  to  be  published,  that  treat 
special  subclasses  of  a  large  class  of  bilinear  models  that  were  introduced  by  Rugh  (1981),  as 
well  various  specializations  of  feedforward  and  feedback  shunting  models  previously  investigated 
by  Sperling  &  Sondhi  (1968),  Pece  et  al.  (1990),  Pinter  (1983,  1989),  Grossberg  (1988),  and 
Yasui  (1982). 

(2)  Cross  kernel  utilization:  Second,  we  have  developed  methods  which  incorporate  the  Wiener 
cross-kernels,  as  well  as  the  self-kernels,  when  testing  and  parameterizing  multi-input  systems. 
The  cross-kernels  characterize  the  nonlinear  interaction  between  multiple  stimulus  inputs — as 
refiected  in  the  nonlinear  response  of  a  neuron.  These  interactions  are  the  foundation  of  many 
visual  models  including,  for  example,  the  Reichardt  motion  detection  model  (Reichardt,  1961). 
Moreover,  the  cross-kernels  provide  a  far  stronger  constraint  on  models  than  the  self- kernels;  they 
generally  simplify  the  system  identification  process;  and  they  are  critical  to  the  identification  of 
some  systems,  (like  Reichardt  motion  systems),  whose  self- kernels  are  identically  zero. 

The  strong  constraint  offered  by  the  cross-kernels  derives  from  the  observation  that  the 
complete  Ith  order  kernel  of  an  N-input  system  consists  of  I-dimensional  temporal  subparts 
of  which  only  N  are  self-kernel  components  and  the  other  —  N  are  cross-kernel  components. 
Hence  the  self-kernel  components  only  represent  of  the  total  kernel.  Therefore,  for  a 

system  studied  with  100  inputs  falling  on  the  receptive  field  (typical  of  physiological  experiments 
we  perform),  the  cell’s  second-order  kernel  will  be  1  percent  self  kernel  and  99  percent  cross 
kernel.  The  cross  kernel  components  dominate  even  more  as  the  kernel  order  is  further  increased. 
Therefore,  if  there  are  nonlinear  interactions  between  the  inputs  to  a  system,  the  preponderance 
of  data  in  the  system’s  kernels  of  order  two  and  higher  is  contained  in  the  cross-kernels. 

Practical  Applications.  In  a  recent  paper  submitted  to  Vision  Research  (Jacobson,  Gaska, 


Chen,  and  Pollen),  we  make  extensive  use  of  the  cross  kernels  to  test  simple  and  complex 
cells  in  the  striate  cortex  of  the  monkey  for  consistency  with  multi-input  LN  and  LNL  models. 
This  same  paper  also  describes  many  practical  matters  related  to  the  application  of  the  new 
methods  to  noisy  physiological  data.  Such  topics  include  (1)  the  benefits  of  cropping  the  support 
re^on  of  multi-dimensional  kernels  to  eliminate  excess  background  noise  from  regions  outside 
of  the  receptive  field,  (b)  statistical  methods  for  quantifying  the  results  from  structural  tests, 
(c)  epochal  kernel  calculation  methods  for  accurately  estimating  rates  of  eye  drift  from  white 
noise  stimulus-response  data,  and  (d)  a  demonstration  of  the  feasibility  of  relating  structural 
test  indices  to  traditional  indices  that  are  used  for  cell  classification  (we  specifically  describe  the 
empirical  relation  between  the  LN  index  and  the  modulation  index). 

Experimental  results 

A  number  of  major  tasks  had  to  be  completed  in  order  to  obtain  the  results  reported  in 
this  section: 

1)  Integrate  a  new  graphics  workstation  (a  Silicon  Graphics  4D/120  GTX  purchased 
under  funding  from  this  contract)  into  our  experimental  laboratory  facilities.  The 
SG  workstation  was  to  be  used  for  both  white  noise  stimulus  generation  and  presentation  of 
traditional  drifting  bar  and  grating  stimuli. 

2)  Develop  the  software  needed  to  apply  the  new  structural  model  testing  pro* 
cedures  developed  in  this  project  to  the  stimulus-response  data  collected  during 
experiments.  This  software  is  described  in  an  appendix  to  this  report. 

3)  Carry  out  a  series  of  electrophysiological  experiments  to  record  the  responses  of 

cortical  cells  to  monocular  and  binocular  spatiotemporal  white  noise  stimuli,  (as  well 
as  to  traditional  bar  and  grating  stimuli  as  needed  for  comparative  analyses  and  for  classifying 
cells  as  simple  versus  complex).  White  noise  studies  each  ran  anywhere  from  10  minutes  to  2 
hours.  ^ _ 

4)  Estimate  the  first-  and  second-order  spatiotemporal  kernels  of  each  ceU  studied. 
The  low-order  kernels  serve  to  summarize  the  white  noise  stimulus- response  relationship  as 
recorded  during  each  cell  study. 

5)  Perform  data  analyses  to  meet  specific  experimental  objectives  previously  out¬ 
lined  in  the  Research  Objectives  section. 

Overview  of  results.  During  the  present  three-year  contract,  we  have  carried  out  a  series  of 
electrophysiological  experiments  using  monocular  and  binocular  presentations  of  random  bar 
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and  checkerboard  stimuli.  In  these  experiments,  we  have  collected  more  than  100  continuous 
hours  of  white  noise  stimulus- response  data  from  more  than  70  cells  in  VI  and  V2.  Most  cells 
were  also  studied  with  traditional  stimuli  (drifting  bars  and  gratings)  to  facilitate  classifica¬ 
tion  by  conventional  criteria,  and  to  directly  determine  preferred  spatial  frequency,  orientation, 
temporal  frequency,  directionality,  and  other  relevant  characteristics  including  the  degree  of 
end-stopping.  These  data  have  provided  us  with  a  rich  characterization  of  the  monocular  and 
binocular  stimulus-response  relationships  of  a  broad  sample  of  unoriented  cells,  simple  cells,  and 
complex  cells. 

The  stimulus- response  data  from  these  studies  were  initially  used  to  estimate  both  the 
first-  and  second-order  kernels  that  parameterize  the  best  (with  respect  to  the  presented  noise 
stimulus)  linear  and  quadratic  models  for  the  input-output  relationship  of  each  cell.  The  3D 
first-order  kernels  describe,  for  about  10  -  200  distinct  spatial  stimulus  input  positions,  the 
spatiotemporal  impulse  response  of  the  best  linear  model.  As  such,  the  first-order  kernel  data 
characterize  the  transmission  delay  from  the  retinae  to  the  cortex  as  well  as  the  detailed  sub¬ 
structure  and  time  evolution  of  the  spatiotemporal  receptive  field(s)  in  one  or  both  eyes.  The  6D 
second-order  kernels  describe,  for  all  possible  pairs  of  spatial  stimulus  positions  and  temporal 
delays,  the  quadratic  impulse  response  correction  associated  with  the  best  second-order  model. 
Accordingly,  the  second-order  kernels  characterize  the  transmission  delay  from  retinae  to  cortex 
as  well  as  the  detailed  temporal  evolution  of  two-point  nonlinear  interactions  as  they  affect  the 
neuron’s  response.  In  effect,  the  second-order  kernels  summarize  the  results  from  a  complete 
two-point  interaction  study,  analogous  to  the  response  profiles  obtained  in  the  seminal  two-bar 
interaction  studies  carried  out  by  Movshon,  Thompson  and  Tolhurst  (1978b)  which  led  them  to 
a  greatly  improved  understanding  of  complex  cells. 

Preliminary  analyses  of  the  collected  data  have  been  reported  in  abstract  form  (Jacobson, 
Gaska,  Chen  &  Pollen,  1989;  Gaska,  Jacobson,  Chen  &  Pollen,  1989),  and  a  major  paper  is  in 
review  at  Vision  Research  which  describes  our  first  physiological  application  of  the  structural 
testing  methods  described  in  the  previous  section.  A  copy  of  this  paper  entitled  “Structural 
testing  of  multi-input  linear- nonlinear  cascade  models  for  cells  in  macaque  striate  cortex”  was 
previously  sent  to  the  Air  Force  on  January  1, 1992,  and  another  copy  is  included  as  an  appendix 
to  this  report.  The  paper  describes  results  from  quantitative  testing  of  multi-input  feedfOTward 
LN  and  LNL  structural  models  for  simple  and  complex  cells.  Results  from  both  monocular  and 
binocular  cells  are  described  in  the  paper,  but  all  analyses  are  characterizations  of  the  monocular 
properties  with  respect  to  the  dominant  eye.  Full  binocular  model  analyses  of  these  same  cell 
data  are  still  in  progress  and  proceeding  well. 
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Monocular  modeling  results.  With  respect  to  the  monocolar  properties  of  simple  and  com¬ 
plex  cells  in  the  monkey,  we  have  obtained  the  following  results.  First,  we  found  that  VI  ceUs 
can  be  divided  into  two  major  groups  based  on  the  degree  to  which  their  first-  and  second-order 
kernels  are  consistent  with  a  multi-input  LN  (or  LNL)  model;  the  resulting  classification  was 
highly  correlated  with  the  simple  cell  versus  complex  cell  distinction  as  determined  by  using  a 
conventional  modulation  index  computed  from  each  cell’s  responses  to  drifting  gratings.  The 
general  agreement  between  these  measures  validates  the  statistical  methods  we  have  developed 
for  testing  a  cell’s  kernels  against  the  predictions  of  the  LN  and  LNL  structural  models. 

However,  the  structural  model  test  indices  are  substantially  more  informative  than  a  sim¬ 
ple  modulation  index.  First,  they  are  not  specific  to  a  particular  stimulus  orientation,  direction, 
spatial  frequency,  or  temporal  frequency,  but  are  in  general  based  on  a  fine  grain  analysis  of 
dynamic  first-  and  second-order  interactions  throughout  the  receptive  field.  Second,  although 
we  describe  in  the  paper  results  from  tests  of  multi-input  LN  and  LNL  models,  the  approach 
generalizes  to  the  testing  of  arbitrary  models  for  which  kernel  relations  can  be  derived.  We  have 
already  developed  tests  for  a  large  class  of  feedforward  nonlinear  models  (Chen,  Jacobson,  and 
Gaska,  1990),  and  similar  tests  have  been  derived  by  ourselves  and  others  for  nonlinear  models 
that  include  feedback  (e.g.,  Pece  et  ai,  1990). 

Jud^ng  from  their  structural  test  indices,  complex  cells  deviated  more  strongly  from  LN 
behavior  than  did  simple  cells,  a  result  we  certainly  anticipated.  However,  as  described  in  our 
paper  in  review,  no  simple  cell  in  our  sample  was  fully  consistent  with  an  LN  model  even  under 
the  steady-state  stimulus  conditions  employed.  On  average,  only  about  60  percent  of  the  first- 
and  second-order  white  noise  stimulus-response  relation  of  simple  cells  was  consistent  with  LN 
behavior.  Only  20  percent  of  the  simple  cells  we  studied  with  both  white  noise  and  gratings  had 
an  LN  consistency  level  that  was  80%  or  greater. 

The  substantial  deviation  of  many  simple  cells  from  LN  behavior  was  a  surprising  result 
which  only  became  apparent  to  us  as  we  proceeded  to  refine  our  statistical  analysis  of  the  data. 
Most  simple  cells  in  the  macaque  striate  cortex  clearly  include  additional  nonlinearities  in  a 
feedforward  or  feedback  path  that  have  a  stioug  effect  on  cell  responses  even  during  steady- 
state  stimulation  with  spatiotemporal  white  noise. 

Evidence  concerning  the  spatiotemporal  filter  properties  of  the  mechanisms  that  underlie 
the  additional  nonlinearities  is  contained  in  the  first-  and  second-order  kernels  of  these  cells.  Our 
subjective  observations  of  the  kernels  suggest  that  these  nonlinear  mechanisms  have  the  signature 
of  units  that  fail  to  encode  contrast  sign,  possibly  resulting  from  subcortical  nonlineaxities  or, 
perhaps  resulting  from  feedback  from  simple  and/or  complex  cells.  Examples  illustrating  the 
nature  of  the  deviations  from  LN  model  behavior  are  included  in  the  paper  in  review.  Farther 
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consideration  will  be  given  to  the  interpretation  of  such  clues  as  we  continue  to  analyze  our 
existing  database. 

Binocular  modeling  results.  Although  we  are  still  in  the  process  of  analyzing  data  from 
binocular  studies  of  simple  and  complex  cells,  several  conclusions  that  relate  to  our  stated 
objectives  have  already  become  apparent. 

First,  we  discuss  results  that  are  relevant  to  Objective  II  in  the  Reseairch  Objectives  sec¬ 
tion.  Since  as  described  above,  many  simple  cells  deviate  substantially  from  LN  behavior  with 
respect  to  their  monocular  response  properties,  we  should  perhaps  not  be  surprised  to  discover 
similar  deviations  from  LN  behavior  with  respect  to  their  binocular  summation  properties.  This 
has  indeed  proven  to  be  true  for  binocular  simple  cells  that  have  been  analyzed  thus  far.  The 
binocular  LN  model  tests  we  are  using  Involve  the  relationship  of  the  left-  and  right-eye  compo¬ 
nents  of  a  cell’s  first-order  kernel  and  the  cross-eye  component  of  the  cell’s  second  order  kernel. 
The  latter  cross-eye  data  characterizes  much  of  the  nonlinear  interaction  between  inputs  from 
the  two  eyes. 

Unfortunately,  we  are  still  not  far  enough  along  with  these  analyses  of  binocular  simple 
cells  to  determine  whether  monocular  or  binocular  deviations  from  LN  behavior  differ  in  degree. 
Nevertheless,  our  results  to  date  showing  deviation  of  binocular  summation  from  LN  behavior 
clearly  contradict,  to  some  extent,  the  results  obtained  by  Ohzawa  and  Freeman  (1986a)  for 
binocular  simple  cells  in  the  cat,  since  they  concluded  that  dichoptic  summation  was  linear 
prior  to  truncation  at  the  output  of  binocular  simple  cells.  Indeed,  even  our  monocular  results 
would  seem  to  contradict  past  studies  of  the  linearity  of  monocular  summation  by  simple  cells 
is  the  cat  (Movshon,  Thompson  and  Tolhurst,  1978a;  Andrews  and  Pollen,  1979;  Pollen,  Gaska 
and  Jacobson,  1987). 

Quite  possibly,  results  using  stimuli  with  a  narrow  spatial-frequency  passband  fail  to 
invoke  certain  types  of  nonlinear  behavior  that  would  become  apparent  with  broadband  stim¬ 
ulation  such  as  is  characteristic  of  spatiotemporal  white  noise.  This  is  not  an  unreasonable 
hypothesis  since  nonlinear  interactions  between  paired  gratings  of  different  spatial  frequencies, 
believed  to  be  of  cortical  origin,  are  well  established  in  the  cat,  and  may  also  be  present  in 
the  cortex  of  the  monkey,  although  the  monkey  has  been  studied  far  less  in  this  regard.  Al¬ 
ternatively,  the  strong  local  contrast  of  white  noise  stimuli  may  strongly  evoke  nonlinearities 
at  subcortical  levels  since,  for  example,  LGN  cells  are  known  to  exhibit  substantial  response 
truncation  at  higher  contrasts.  It  is  presently  not  known  whether  nonlinearities  arising  in  the 
separate  ON  and  OFF  channels  of  the  LGN  are  effectively  compensated  for  by  mechanisms  at 
the  cortical  level. 
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In  either  case,  LGN  nonlinearities  can’t  explain  our  observed  deviations  from  LN  behavior 
with  respect  to  dichoptic  summation.  This  conclusion  is  reached  as  follows.  First,  we  know  that 
the  initial  point  of  convergence  of  dichoptic  signals  onto  single  neuronal  target  cells  occurs 
within  the  cortex.  But,  in  general,  if  two  channels  are  combined  linearly  at  some  stage,  then  the 
cross-channel  kernel  measured  at  that  stage  must  be  identically  zero  even  for  highly  nonlinear 
input  channels.  In  the  present  context,  nonlinear  subcortical  channels  could  not  lead  to  nonzero 
cross-eye  kernel  unless  a  cortical  nonlinearity  follows  convergence  of  the  dichoptic  signals  at  the 
cortical  level.  This  certunly  bolsters  the  argument  in  favor  of  a  cortical  origin  for  the  apparent 
failure  of  dichoptic  summation  during  white  noise  stimulation. 

With  regard  to  Objective  III,  we  strongly  believe  that  we  have  found  two  classes  of 
complex  cells  that  are  the  monkey  equivalents  to  the  two  classes  of  binocular  complex  cells 
discovered  by  Ohzawa  and  Freeman  (1986b)  in  the  cat.  They  refer  to  these  as  phase-specihc 
and  non  phase-specific  complex  cells,  as  describes  their  sensitivity  to  the  relative  phas'^  of 
dichoptically  presented  pairs  of  gratings. 

Although  dichoptic  grating  stimuli  have  not  been  specifically  used  in  the  present  investi¬ 
gation,  analysis  of  our  binocular  data  has  demonstrated  that  two  subclasses  of  binocular  complex 
cells  exist  also  in  the  monkey  visual  cortex.  As  anticipated  in  the  grant  application,  one  subclass 
(presumed  to  be  phase-specific)  have  non-zero  second-order  cross-eye  kernels — hence,  as  argued 
above,  a  fundamental  nonlinearity  occurs  after  the  convergence  of  inputs  from  the  two  eyes.  In 
contrast,  a  second  subclass  of  complex  cells  (presumed  to  be  phase  non-specific)  have  second- 
order  cross-eye  kernels  that  are  identically  zero — hence  all  nonlinearities  must  occur  before  the 
convergence  of  inputs  from  the  two  eyes. 

A  binocular  complex  cell  whose  second-order  cross-eye  kernel,  (as  well  as  all  other  higher- 
order  cross-eye  kernels),  is  identically  zero  would  be  insensitive  to  the  phase  of  a  pair  of  dichopti¬ 
cally  presented  gratings.  This  fact  supports  the  investigator’s  presumption  that  that  the  second 
subclass  of  complex  cells  described  in  the  above  paragraph,  having  a  zero  cross  kernel,  is  likely 
to  correspond  to  the  phase  non-specific  subclass  identified  by  Ohzawa  and  Freeman  (1986b).  K 
this  association  is  correct,  then,  as  discussed  above,  phase  independence  of  this  class  of  complex 
cell  is  a  straightforward  consequence  of  the  structure  of  the  underlying  network. 

However,  as  discussed  in  Objective  V,  Ohzawa  and  Freeman  (1986b)  also  suggested  an 
alternative  model  for  non-  ihase  specific  complex  cells  in  which  a  lack  of  phase  specificity  to  pairs 
of  extended  gratings  could  result  from  a  random  jitter  of  binoctilar  disparity  tuning  throughout 
the  receptive  field.  However,  a  random  jitter  model  predicts  a  non-zero  second-order  cross-eye 
kernel  whose  binocular  interaction  phase  varies  randomly  across  space.  But,  of  the  binocular 
complex  cells  with  non-zero  second-order  cross-kernels  studied  by  the  present  investigators. 
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none  exhibited  such  randomness  in  their  interaction  phases.  Rather,  they  exhibit  an  essentially 
constant  (non-random)  interaction  phase  which  would  lead  to  phase-specihc  responses  to  pairs 
of  dichoptically  presented  gratings.  (This  is  apparent  from  the  stated  properties  of  the  kernel 
together  with  the  form  of  the  multi-input  Wiener  G-functional  expansion  term  of  second  order 
which  can  together  be  used  to  predict  the  response  contribution  of  the  cross-eye  kernel).  In 
short,  a  random  phase  model  for  phase  non-specific  complex  cells  in  the  monkey  is  rejected  by 
our  data.  This  therefore  answers  Objective  V  which  sought  to  determine  the  proper  model  for 
phase  non-specific  cells. 

With  respect  to  Objective  IV,  the  feasibility  of  modeling  phase- specific  cells  in  the  mon¬ 
key  using  a  straightforward  binocular  generalization  of  the  Pollen  and  Ronner  (1981)  4-cell 
quadrature  model,  our  results  clearly  show  that  this  is  a  reasonable  model.  In  particular,  the 
second-order  cross-eye  kernels  have  essentially  the  same  appearance  as  the  second-order  same- 
eye  kernels,  consistent  with  a  straightforward  binocular  generalization  of  the  monocular  4-cell 
model  by  supposing  that  the  underlying  simple  cells  are  binocular.  Of  course,  Ohzawa,  DeAnge- 
lis,  and  Freeman  (1990a)  have  already  arrived  at  the  same  conclusion  based  on  their  experiments 
employing  white  noise  to  study  binocular  encoding  by  complex  cells  in  the  cat. 

With  respect  to  Objective  VI,  we  have  analyzed  enough  first-order  kernels  from  binoc¬ 
ular  simple  cells  to  conclude  that  the  restriction  to  0  and  180  degree  relative  phase  relations 
for  the  left-  and  right-eye  spatial  receptive  field  profiles  predicted  by  our  disparity  estimation 
theory  (Gaska,  et  ai,  1988)  must  be  wrong.  The  relative  phases  of  binocular  simple  cells  in  the 
monkey  are  more  variable  than  our  model  would  predict.  The  variability  of  the  binocular  phase 
difference  was  first  emphasized  by  Ohzawa,  DeAngelis  and  Freeman  at  ARVO  (1990)  where  we 
also  presented  our  first  preliminary  data  on  the  relative  phases  of  binocular  filters  in  the  monkey 
(Jacobson  and  Gaska,  1990). 

In  retrospect,  given  our  familiarity  with  the  seminal  findings  of  Poggio  and  Fischer  (1977) 
on  near,  far,  tuned  inhibitory,  and  tuned  excititory  binoctilar  mechanisms  in  the  cortex,  we 
should  have  ourselves  anticipated  widely  varying  phase  relations  between  the  left-  and  right- 
eye  filters  of  binocular  simple  cells.  It  is  interesting  to  note  that  binocular  simple  cells  whose 
binocular  filters  have  a  0  or  180  degree  phase  relation  could  operate  directly  on  a  sum  or  difference 
image  representation  derived  by  mechanisms  presynaptic  to  the  binocular  simple  cell.  However, 
binocular  cells  with  more  general  phase  relations  must,  in  effect,  apply  independent  filters  to 
the  inputs  from  the  two  eyes. 

Finally,  with  regard  to  Objective  VH,  the  simple  cell  component  of  our  original  model 
for  cortical  disparity  estimation  must  clearly  be  modified  to  incorporate  the  results  on  relative 
inter-eye  phase  variability  described  above.  Such  binocular  phase  results  obtained  first  (for  the 
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cat)  in  Freeman’s  laboratory  have  led  them  (Ohzawa  et  al,  1990a,b,  DeAngelis  et  ai,  1991)  and 
Nomura  et  al  (1990)  to  independently  suggest  that  the  filter  envelopes  of  all  simple  cells  may 
be  in  retinal  correspondence  (ie.  have  no  spatial  dispsirity  shift)  and  that  disparity  tuning  may 
result  solely  from  relative  phase  differences  in  the  left  and  right  eye  filters.  This  represents  a 
radical  departure  from  previous  binocular  theories  in  which  position  differences  were  emphasized 
as  a  basis  for  disparity  tuning.  As  shown  by  Nomura  et  al  (1990),  phase  differences  in  the  left 
and  right  eye  filters  of  simple  cells  certainly  have  a  profound  influence  on  their  disparity  tuning 
curves,  producing  near,  far,  tuned  excititory,  and  tuned  inhibitory  responses  such  as  those 
described  by  Poggio  and  Fischer  (1977)  for  real  monkey  visual  cells.  However,  we  have  recently 
developed  an  alternative  interpretation  of  the  inter-eye  phase  variations  which  is  motivated  by 
functional  considerations. 

Briefly,  complex-like  units  in  our  modified  theory  for  disparity  estimation  encode  not  only 
the  in-phase  component  of  inter-eye  Gabor  signal  correlation  (tuned  inhibitory  or  tuned  exciti¬ 
tory  response  behavior)  as  in  our  original  theory,  but  also  a  quadrature  correlation  component 
(near  or  far  response  behavior).  This  is  an  important  point.  In  effect,  a  phase-dependent 
complex  cell  in  the  cortex  (we’re  talking  about  real  cells  here!)  effectively  acts  to  encode  a  cross- 
correlation  between  two  complex-valued  spectral  coefficients  (Gabor  or  whatever),  each  derived 
with  respect  to  a  different  eye.  Mathematically,  such  a  correlation  only  yields  a  real-valued 
result  if  the  spectral  coefficients  are  identical  (such  as  when  the  left  and  right  eye  image  patches, 
as  seen  by  the  cell,  are  non-disparate);  otherwise  the  correlation  result  is  complex-valued.  If 
the  entire  correlation  is  to  be  encoded,  then  there  must  be  both  in-phase  and  quadrature  mech¬ 
anisms  to  encode  the  real  part  and  imaginary  part  of  the  correlation,  respectively.  Returning 
again  to  a  discussion  of  our  modified  theory,  the  addition  of  the  new  quadrature  correlation 
component  therefore  causes  the  binocular  correlation  representation  to  become  informationally 
complete.  It  can  also  help  to  control  vergence  movements,  and  can  be  used  together  with  the 
in-phase  correlation  component  to  normalize  network  activity  to  discount  contrast  differences  in 
the  two  eyes — something  not  possible  in  our  original  theory. 

The  completeness  of  the  new  binocular  representation  relates  to  the  fact  that  the  response 
of  a  binocular  mechanism  (simple  or  complex  cell)  with  an  arbitrary  inter-eye  phase  relation  can 
be  obtained  as  a  linear  combination  of  binocular  in-phase  and  quadrature-phase  mechanisms. 
Therefore,  a  considerable  redundancy  of  information  is  inherent  in  most  theories  that  postulate 
that  disparity  information  is  encoded  by  a  population  of  cells  (>  2)  that  differ  oitly  in  their 
inter-eye  phase  differences.  In  practice,  any  linear  combination  of  the  in-phase  and  quadrature- 
phase  components  could  be  used  a  representational  basis  for  the  inter-eye  correlation  function. 
Local  orthogonality  of  inter-eye  phase  differences  is  predicted  by  the  modified  theory  for  a  small 
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cluster  of  cells  that  would  encode  the  binocular  correlation  at  a  single  position,  orientation  and 
spatial-frequency.  However,  the  choice  of  basis  could  still  vary  across  the  cortex,  consistent  with 
the  empirical  finding  that,  when  data  are  pooled  across  all  cells,  the  inter-eye  phase  differences 
of  simple  cells  appear  to  be  distributed  quite  uniformly. 

Freeman’s  group  and  Nomura  et  ai,  (1990)  as  discussed  above,  are  promoting  the  idea 
that  disparity  estimation  might  be  achieved  strictly  on  the  basis  of  mechanisms  whose  left  and 
right  eye  receptive  field  envelopes  are  in  strict  retinal  correspondence. 

In  contrast,  our  modified  model  for  disparity  estimation  still  retains  spatial  displacements 
of  the  left-  and  right-eye  receptive  fields  to  estimate  non-zero  disparities  in  an  image.  Phase 
differences  in  the  left-  and  right-eye  spatial  filters  of  simple  cells  would  only  be  present  to  permit 
the  encoding  of  both  in-phase  and  quadrature-phase  inter-eye  correlation  components  at  a  single 
disparity — providing  therefore  a  parsimonious  explanation  for  the  existence  of  tuned  excititory, 
tuned  inhibitory,  near,  and  far  disparity-tuned  cells.  However,  the  presence  of  such  mechanisms 
would  in  no  way  negate  the  importance  of  using  displaced  cortical  filters  as  a  means  for  encoding 
non-zero  disparities. 

Unfortunately,  as  pointed  out  by  DeAngelis  et  ai  (1991),  there  is  still  insufficient  empir¬ 
ical  evidence  to  decide  between  a  pure  phase  model  (such  as  they  are  promoting)  or  a  combined 
displacement  and  phase  model  (such  as  ours)  for  binocular  Altering  by  simple  cells.  This  very 
fundamental  question  concerning  the  encoding  of  disparity  certainly  ranks  as  one  of  the  most 
important  unresolved  issues  facing  the  cortical  vision  community  today. 
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Quantitative  procedures  were  developed  for  testing  block-structured  models  for  multi¬ 
input  nonlinear  visual  circuits  studied  with  spatiotemporal  white  noise.  The  new  methods  were 
used  to  test  multi-input  structural  models  for  the  monocular  and  binocular  filtering  by  simple 
and  complex  cells  in  the  striate  cortex  of  the  macaque  monkey.  A  linear-nonlinear  (LN)  model 
test  index  was  found  to  be  suitable  for  classifying  cells  as  simple  versus  complex.  Although 
simple  cells  were  better  modeled  as  LN  systems  than  complex  ceDs,  most  simple  cells  deviated 
considerably  from  LN  behavior.  A  nonlinearity  of  cortical  origin  would  appear  to  be  responsible, 
possibly  activated  more  strongly  by  broadband  noise  than  by  sinewave  grating  stimuli.  Also,  two 
classes  of  binocular  complex  cells  were  identified.  Whereas  all  binocular  complex  cells  necessar¬ 
ily  have  a  non-zero  second-order  “same-eye’  interaction  kernel,  their  second-order  “cross-eye” 
interaction  kernel  could,  it  was  found,  be  either  non-zero  or  identically  zero.  The  two  binocu¬ 
lar  complex  cell  classes  are  thought  to  be  the  monkey  equivalents  of  the  “phase  specific”  and 
“phase  non-specific”  cells  previously  discovered  by  Ohzawa  and  Freeman  (1986b)  in  the  cat. 
A  random-jitter  disparity  tuning  model  suggested  by  Ohzawa  and  Freeman  was  found  to  be 
inconsistent  with  “phase  non-specific”  cells  in  the  monkey,  rather  they  must  linearly  combine 
nonlinear  monocular  inputs — as  alternatively  suggested  by  Ohzawa  and  Freeman.  The  left  and 
right  eye  filters  of  simple  cells  in  the  monkey  were  found  to  have  broadly  varying  phase  relar 
tionships.  Finally,  a  quadrature  correlation  model  for  stereo  depth  estimation  was  developed  to 
account  for  recent  new  findings  by  ourselves,  and  others. 
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APPENDIX:  Software  Description 

ND  :  A  C'callable  function  library  for  multi-dimensional  array  processing. 

A  library  of  ‘C’  callable  functions  has  been  developed  to  facilitate  the  storage  and  process¬ 
ing  of  multi-dimensional  arrays  of  data.  This  function  library,  denoted  hereafter  as  “ND”,  enables 
one  to  manipulate  arrays  with  arbitrarily  many  dimensions,  including  also  zero- dimensional  ar¬ 
rays  (ie.  scalars). 

Each  array  in  ND  can  contain  data  of  type  “int”,  “float”,  “double”,  “complex  int”,  “com¬ 
plex  float”,  or  “complex  double.”  In  addition,  an  “array”  data  type  is  included  to  permit  arrays 
of  arrays  of  ...  to  any  depth.  Functions  are  also  included  that  permit  the  user  to  dynamically 
add  new  data  types  which  can  then  be  allocated,  deallocated,  and  otherwise  manipulated  by 
many  of  the  basic  ND  functions. 

Various  ND  functions  can  be  used  to  create  child  arrays  that  define  alternative  multi¬ 
dimensional  mappings  into  all  or  part  of  an  existing  parent  array’s  data  field.  Any  child  array 
can  also  have  children  of  its  own  which  continue  to  refer  to  the  original  ancestral  array’s  data 
field.  Trees  of  array  dependencies  are  automatically  maintained  by  link  fields  so  that  a  parent 
array  with  “living”  children  can  be  deallocated  by  the  user  at  any  time;  such  a  parent  array  is 
simply  flagged  as  dormant  and  the  actual  deallocation  of  the  parent  array’s  data  is  automatically 
deferred  until  all  of  its  descendent  arrays  have  themselves  been  deallocated. 

ND  allows  the  user  to  “attach”  an  array  of  binary  data  located  in  a  file  on  a  system  disk; 
the  data  in  the  file  is  mapped  into  the  user  process’s  virtual  memory  space  and  the  data  in  the 
file  can  then  be  read/written  or  both  like  any  other  ND  array  that  resides  in  main  memory.  The 
file  is  automatically  unmapped  and  closed  when  the  attached  array  is  deallocated. 

ND  currently  includes  functions  for:  element-by-element  sums  and  products  of  arrays, 
outer  sums  and  products  of  arrays,  inner  products  of  arrays,  element-by-element  polynomial  ex¬ 
pansion  of  an  array,  array  smoothing  and  integration,  array  interpolation  and  decimation,  array 
transpositions,  array  duplication  and  copying,  subarray  mappings  and  data  extractions,  array 
data  type  conversions,  array  storage  and  retrieval  from  disk,  array  statistics  calculation,  and 
multi-dimensional  array  index  manipulations.  All  functions  minimally  accept  array  arguments 
whose  data  type  conforms  to  one  of  the  six  fundamental  real  and  complex  data  types  mentioned 
earlier.  When  a  function  receives  two  or  more  array  arguments  with  different  data  types,  auto¬ 
matic  t}rpe  conversion  is  performed  as  necessary  on  the  inputs  and  the  function  returns  a  result 
whose  data  type  is  automatically  promoted. 

The  ND  function  library  exemplifies  a  programming  philosophy  that  is  crucial  to  the 
rapid  prototyping  and  development  of  multi- dimensional  signal  processing  applications.  Namely, 
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rather  than  implementing  an  algorithm  (e.g.,  an  FFT)  as  multiple  routines,  each  specific  to  data 
of  a  different  array  dimensionality  and  type,  one  should  instead  seek  to  write  a  sinf^e  routine 
that  is  applicable  to  arrays  of  arbitrary  dimensionality  and  data  type.  With  careful  design,  such 
generality  can  be  achieved  with  little  sacrifice  in  computational  efficiency. 

The  ND  C-callable  function  library  serves  as  the  basis  for  most  data  analysis  software 
that  was  developed  during  the  present  project.  We  next  describe  a  module  that  builds  on  the 
capabilities  of  the  ND  function  library  to  make  it  possible  to  interactively  view  ND  arrays. 

Although  the  ori^nal  and  most  capable  version  of  the  ND  library  was  developed  to  run  on 
Silicon  Graphics  workstations,  a  portable  version  of  the  ND  library  with  most  of  the  capabilities 
of  the  original  version  is  also  now  available.  The  portable  version  is  suitable  for  use  on  PCs  or 
any  other  platform  on  which  a  C  compUer  is  available. 

LOOK  :  A  C-callable  software  module  for  interactive  viewing  of  ND  arrays. 

As  previously  discussed,  the  nonlinear  network  analysis  methods  developed  during  this 
project  make  use  of  estimated  first-  and  second-order  kernels  of  a  neuron.  For  a  2D  spatial 
noise  study,  the  first  order  kernel  is  a  three-dimensional  function  and  the  second-order  kernel 
is  a  six- dimensional  function  (See  Fig.  1).  These  functions  must  be  viewed  and  interactively 
manipulated  by  the  experimenter  at  several  stages  of  analysis. 

For  example,  the  second-order  kernel  of  a  neuron  is  generally  non-zero  only  over  a  limited 
repon  of  its  six- dimensional  space.  Only  by  viewing  the  computed  kernel  estimate  can  the 
experimenter  determine  the  actual  location  and  extent  of  this  6D  region.  In  some  cases,  the 
estimated  kernel  is  observed  to  be  so  corrupted  by  noise  that  it  must  be  completely  discarded. 
However,  when  the  kernel  is  discriminable  from  the  noise,  the  experimenter  must  determine 
whether  the  kernel  power  is  wholly  contained  within  the  re^pon  over  which  the  estimate  was 
computed.  If  not,  the  re^on  of  computation  must  be  expanded  and  the  kernel  recmnputed. 
Once  the  full  support  of  the  kernel  is  included  in  the  estimate,  the  experimenter  must  then  crc^ 
the  kernel  in  six  dimensions  so  as  to  eliminate  re^ons  consisting  entirely  of  noise.  Such  cropping 
greatly  enhances  the  sensitivity  of  the  methods  we  have  developed  to  evaluate  various  nonlinear 
models  for  a  neuron’s  network. 

Also,  with  a  little  practice,  one  acquires  the  ability  to  infer  many  prc^erties  of  a  neuron 
by  merely  viewing  its  kernels.  For  example,  it  becomes  quite  easy  to  subjectively  determine 
from  a  neuron’s  kernels  whether  it  is  orientation  tuned,  whether  it  is  directionally  selective,  or 
whether  it  should  be  classified  as  simple  or  complex. 

To  make  these  interactive  tasks  possible,  a  C-callable  software  module,  hereafter  referred 
to  as  “LOOK”,  has  been  developed  during  this  project.  LOOK  enables  the  experimenter  to 
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view  a  multi-dimensional  array  of  data  with  arbitrarily  many  dimensions.  The  user  can  view 
all  dimensions  simultaneously  or,  alternatively,  view  the  data  contained  within  an  arbitrary 
rectangular  subvolume  of  the  multi-dimensional  space. 

Multi-dimensional  data  viewing  is  accomplished  in  LOOK  by  first  partitioning  an  N- 
dimensional  data  array  into  three-dimensional  rectangular  subarrays,  and  then  recursively  nest¬ 
ing  these  blocks  in  3D  space  so  as  to  encompass  the  remaining  input  dimensions.  The  resulting 
3D  volume  mosaic  image  can  then  be  viewed  one  2D  (z,y)  slice  at  a  time  on  the  computer 
screen,  using  a  scroll  bar  to  move  through  the  third  (z)  dimension. 

The  user  can  use  LOOK  to  manipulate  the  viewed  data  in  several  ways.  First,  the  original 
ND  array  can  be  interactively  cropped  to  include  any  desired  rectangular  subvolume  of  data. 
Second,  each  dimension  (l,...,iV)  of  the  cropped  data  array  can  be  independently  assigned  to 
one  of  the  z,y,  or  z  dimensions  of  the  volume  mosaic  image.  Third,  one  can  specify  the  desired 
nesting  order  of  the  input  dimensions  assigned  to  each  output  dimension  (x,y,z)  of  the  volume 
mosaic  image.  Optionally,  all  input  dimensions  can  be  assigned  to  output  dimensions  x  and  y, 
enabling  the  experimenter  to  instantaneously  view  the  entire  ND  array  on  the  screen  in  the  form 
of  a  recursively  nested  mosaic  of  two-dimensional  images.  Finally,  LOOK  permits  the  user  to 
save  any  cropped  subregion  of  an  ND  array  to  a  new  file,  and  both  2D  and  3D  contour  plots 
can  be  interactively  generated  from  the  data  in  the  LOOK  window. 

LOOK  was  invaluable  to  this  project.  It  provided  a  window  into  the  operation  of  kernel 
analysis  software  during  development  and  debugging,  and  it  is  extensively  used  now  to  perform 
routine  data  analysis  tasks.  (An  example  of  the  appearance  of  the  LOOK  user  interface  is  shown 
in  the  figure  attached  at  the  end  of  this  report.  Illustrated  in  the  LOOK  data  window  in  that 
figure  is  a  mosaic  view  of  all  six  dimensions  of  the  second-order  kernel  of  a  simple  cell  from 
monkey  VI.)  LOOK  currently  runs  on  Silicon  Graphics  Workstations  using  the  SGI-standard 
GL  graphics  library. 

KCAL:  Nth-order  monocular  and  binocular  kernel  calculation. 

Another  application  developed  during  this  project  is  a  program  for  kernel  calculation, 
hereafter  referred  to  as  “KCAL”.  KCAL  is  used  to  compute  the  self-  and  cross-kernels  of  multi¬ 
input  visual  neurons  driven  through  either  one  or  both  eyes.  Owing  to  its  use  of  the  ND  function 
library,  KCAL  can  be  employed  to  estimate  system  kernels  of  any  order.  To  use  KCAL,  the 
experimenter  must  specify  the  names  of  stimulus  and  response  description  files,  and  the  root 
name  of  the  output  kernel  file(s).  The  user  is  then  prompted  for  additional  information  including 
the  order  of  the  desired  kernel,  and  the  region  over  which  the  kernel  is  to  be  computed.  The 
region  of  computation  is  specified  by  a  time-delay  interval  together  with  a  rectangular  spatial 


21 


support  re^on  that  is  separately  specified  for  each  eye. 

KCAL  can  be  used  to  compute  kernels  associated  with  binary,  ternary,  or  general  N-ary 
stimuli.  Kernels  can  also  be  computed  with  respect  to  modulation  along  multiple  axes  in  color 
space,  including  nonlinear  response  interactions  arising  between  stimulus  components  directed 
along  these  color  axes.  KCAL  also  provides  an  option  to  compute  multiple  estimates  of  the 
system  kernels,  each  computed  from  a  distinct  epoch  of  the  stimulus  response  record.  This  capa¬ 
bility  is  currently  used  to  assess  the  degree  of  stationarity  in  each  neuron’s  response  properties. 
In  particular,  conditions  such  as  significant  eye  movement  or  loss  of  spike  discrimination  are 
readily  detectable  on  the  basis  of  systematic  changes  in  such  epochal  kernel  estimates. 

To  facilitate  rapid  calculation  of  higher  order  kernels,  KCAL  automatically  detects  the 
number  of  CPUs  in  the  host  computer  (any  Silicon  Graphics  Power  Series  Multi- processor  ma^ 
chine)  and  spawns  additional  threads  to  distribute  the  computation  among  all  available  CPUs. 
(The  computer  used  in  this  project  was  a  Silicon  Graphics  4D/120  GTX  which  has  two  CPUs). 

subheadingNonlinear  structure  testing  software 

The  last  category  of  software  includes  programs  that  are  used  to  test  candidate  structural 
models  for  a  neuron’s  underlying  multi-input  nonlinear  network.  Each  structural  model  (e.g.,  a 
multi-input  L-N  or  L-N-L  model)  is  associated  with  one  or  more  necessary  conditions.  These 
necessary  conditions  are  multi-  dimensional  equations,  (involving  the  system  kernels),  that  must 
hold  if  the  associated  structural  model  is  valid. 

In  practice,  each  necessary  condition  is  evaluated  in  two  principal  st^es:  First,  a  sys¬ 
tem’s  estimated  kernels  must  be  substituted  into  an  equation  that  defines  the  necessary  con¬ 
dition.  Second,  statistical  methods  must  be  employed  to  test  the  hypothesis  that  the  resulting 
multi-dimensional  equation  is  satisfied.  If  this  hypothesis  is  rejected  by  the  data,  then  the  cor¬ 
responding  structural  model  is  also  rejected;  otherwise,  the  corresponding  structural  model  is 
deemed  to  be  consistent  with  the  kernel  data. 

Software  has  been  developed  to  generate  each  multi- dimensional  term  comprising  the  nec¬ 
essary  conditions  associated  respectively  with  the  multi-input  L-N  and  L-N-L  models.  (Coverage 
of  additional  models  is  in  the  works).  The  same  software  also  resolves  a  free  parameter  (a  scale 
factor)  in  each  resulting  equation  by  employing  a  minimum  mean  square  error  (MMSE)  crite¬ 
rion.  There  results  a  multi-dimensional  error  residual  which  is  retained  for  subsequent  statistical 
analysis. 

Software  has  also  been  written  to  estimate  the  statistics  of  the  multi-dimensional  noise 
process  that  corrupts  each  term  in  the  above-mentioned  equations.  Estimation  of  the  nmse 
process  statistics  is  facilitated  by  computing  estimates  of  the  system  kernels  over  non-causal  time 
intervals  where  the  signal  component  is  necessarily  zero.  The  estimated  noise  statistics  also  take 
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into  account  the  fact  that  the  necessary  conditions  generally  involve  nonlinear  transformations  of 
the  kernels,  resulting  therefore  in  a  nonstationary  noise  process.  The  term-by-term  noise  statistics 
are  subsequently  combined  to  produce  an  estimate  of  the  statistics  of  the  multi-dimensional  noise 
process  that  corrupts  the  residual  obtained  during  evaluation  of  each  necessary  condition. 

Finally,  software  has  been  developed  to  apply  several  statistical  tests  to  each  multi¬ 
dimensional  residual.  In  particular,  hypothesis  testing  is  used  to  determine  if  the  difference 
between  each  residual  and  its  corresponding  estimated  noise  process  is  too  large  to  be  accounted 
for  by  chance.  If  the  differences  are  too  great,  then  the  multi- dimensional  residual  is  judged 
to  contain  a  systematic  error.  This  implies  that  the  corresponding  necessary  condition  is  not 
satisfied,  and  hence  the  associated  structural  model  must  be  rejected.  Otherwise,  the  structural 
model  is  deemed  to  be  consistent  with  the  estimated  low-order  kernels  of  the  neural  system 
under  study.  As  the  spatiotemporal  resolution  and  signal-to-noise  ratio  of  the  kernel  estimates 
improve,  increasingly  precise  model  discriminations  can  be  made. 

Readers  of  this  report  who  have  an  interest  in  obtaining  any  of  the  software  described 
above  should  feel  free  to  contact; 

Lowell  D.  Jacobson 

Department  of  Neurology 

University  of  Massachusetts  Medical  School 

55  Lake  Avenue  North 

Worcester,  MA  01655 

Telephone:  508  856-4147 
Email:  lowell@ummed.edu 

Note:  The  ND  library  is  available  in  a  portable  version  that  is  in  a  ready- to-ship  state,  although 
documentation  is  still  somewhat  sparse.  LOOK,  currently  specific  to  Silicon  Graphics  platforms, 
will  probably  go  through  a  major  revision  in  the  forseeable  future  to  make  it  compatible  with  X 
window  enviornments.  A  portable  version  of  KCAL  could  be  developed  in  a  short  time  if  there  is 
a  demand  for  it.  The  structural  test  software  is  the  least  refined  and  is  not  entirely  self-contained 
since  it  uses  some  functions  from  the  ‘^Recipes  in  C”  library. 
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Figure  1.  Appearance  of  the  LOOK  user  interface  As  was  described  in  the  Appendix, 
LOOK  was  developed  to  facilitate  interactive  viewing  of  data  arrays  with  arbitrarily  many 
dimensions.  Illustrated  in  the  LOOK  window  above  are  all  six  dimensions  of  a  (10  x  8  x  3  x  10 
X  8  X  3)  slice  CUT  from  the  second-order  kernel  estimated  for  a  simple  cell  from  monkey  VI. 

The  six  dimensions  shown  in  the  LOOK  window  correspond  to  the  (2D)  spatial  position 
and  (ID)  delay  of  test  (d0,dl,d2)  and  reference  (d3,d4,d5)  stimulus  spots;  the  stimulus  spots 
constitute  a  two-dimensional,  60  Hz  noise  movie  that  was  presented  on  a  10*8  grid  which  covered 
the  cell’s  receptive  field.  The  interpretation  of  the  data  is  as  follows.  At  (6D)  coordinate  where 
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the  kernel  appears  bright,  paired  stimuli  caused  net  excitation  (inhibition)  of  the  cell’s  response 
when  the  spots  were  of  the  same  (different)  polarity.  Conversely,  at  coordinates  where  the  kernel 
appears  dark,  paired  stimuli  caused  net  inhibition  (excitation)  when  the  spots  were  of  the  same 
(different)  polarity.  [Note:  Same  polarity:  B-B  or  D-D,  Different  polarity:  B-D  or  D-B,  where  B 
=  bright  spot,  D  =  dark  spot  relative  to  mean  background  luminance.]  The  excititory /inhibitory 
sensitivity  of  the  cell  to  paired  spots,  as  revealed  by  the  second-order  kernel,  is  not  aU  or  nothing, 
but  rather  is  continuously  distributed. 

Note  that  via  the  SELECT  and  ORDER  control  windows,  the  user  has  instructed  LOOK 
to  display  all  six  array  dimensions  simultaneously  in  the  form  of  ‘*a  2D  mosaic  of  2D  mosaics 
of  2D  images.”  The  innermost  x  -  y  dimensions  in  the  LOOK  window  correspond  to  dO  -  d\ 
(spatial  coordinate  of  test  spot),  the  next  outer  x  —  y  dimensions  are  dZ  —  dA  (spatial  coordinate 
of  reference  spot),  and  the  outermost  x  —  y  dimensions  are  d2  —  d5  (time  delays  since  presentation 
of  test  and  reference  spots).  The  spatial  and  temporal  resolution  of  the  illustrated  kernel  are  0.1 
deg/sample  and  16.7  msec/sample,  respectively. 


